07-12-2021

Statistical reporting quality

Importance:

  • Reproduction and critical assessment
  • Erroneous results and misinformation
  • Standardization

Many aspects already studied in (among others) psychology, but not in sociology.

Topics of interest

  1. Requested adherence to statistical reporting guidelines (i.e., APA manual) by journals
  2. Statistical reporting errors
    • inconsistencies
    • gross inconsistencies
  3. Publication bias/p-hacking
    • replication Gerber and Malhotra (2008)
  4. ‘Bump’ in p-values
  5. Marginal significance

Note that not all topics are (fully) addressed in this presentation.

Data sets (1)

  • Data set with information on which journals require adherence to which guidelines
    • all 143 sociology journals from Clarivate Analytics’ Web of Science (2016) were included
  • Data set AllP
    • automatic retrieval
    • all p-values retrieved using Epskamp and Nuijten (2016)’s statcheck
    • information on marginal significance

Data sets (2)

  • Data set APA
    • automatic retrieval
    • all APA-reported results as retrieved using statcheck
  • Data set Hyp
    • Created to replicate Gerber and Malhotra (2008)
    • manual retrieval
    • all p-values of hypotheses
    • all reproducible results of hypotheses
    • information on marginal significance for results of hypotheses

Data sets (3)

Small problem when running statcheck

For JMF, statcheck could initially not extract results with a minus sign. One of the package’s authors solved this problem for us by adapting the GitHub version of the package, which we installed using the R code below.

install.packages("devtools")
devtools::install_github("MicheleNuijten/statcheck")
install.packages("statcheck")

Hyp article selection process

Statistical analyses

  • For all topics: frequencies & percentages
  • Logistic regressions for testing hypotheses

\[\scriptsize \begin{align} H1: log(Inconsistency_{i}) &= \beta_0 + \beta_1\times hypothesis_i + \epsilon_i \\ H2: log(Gross\,inconsistency_{i}) &= \beta_0 + \beta_1\times hypothesis_i + \epsilon_i \\ H3: log(Publication\,bias_{i}) &= \beta_0 + \beta_1\times hypothesis_i + \epsilon_i \\ H4: log(Marginal\,significance_{i}) &= \beta_0 + \beta_1\times hypothesis_i + \epsilon_i \end{align} \]

Statistical reporting guidelines (1)

13 of 143 journals (9.1%) requested authors to adhere to APA statistical reporting guidelines

Statistical reporting guidelines (2)

Statistical reporting errors

APA (n = 505):
- 69 results (13.7%) were inconsistent
- 8 results (1.6%) were grossly inconsistent

Hyp (n = 351):
- 17 results (4.8%) were inconsistent
- 3 results (0.9%) were grossly inconsistent

Publication bias & bump in p-values (1)

Publication bias & bump in p-values (2)

References

Clarivate Analytics’ Web of Science. 2016. “Journal Citation Reports: Sociology, 2016.” com.proxy.library.uu.nl/JCRJournalHomeAction.

Epskamp, Sacha, and Michele B. Nuijten. 2016. Statcheck: Extract Statistics from Articles and Recompute P Values. https://cran.r-project.org/src/contrib/Archive/statcheck/.

Gerber, Alan S., and Neil Malhotra. 2008. “Publication Bias in Empirical Sociological Research: Do Arbitrary Significance Levels Distort Published Results?” Sociological Methods and Research 37 (1): 3–30.